Orthogonally system arrangements for data center facility

ABSTRACT

A facility is described that includes one or more enclosures defining an interior space, a plurality of power taps, a plurality of coolant supply taps, and a plurality of coolant return taps. A flow capacity of the supply taps and a flow capacity of the return taps can be approximately equal over a local area of the interior space. The plurality of power taps, the plurality of supply taps, and the plurality of return taps can be divided into a plurality of zones, with taps of each zone are configured to be controllably coupled to a power source or a coolant source independently of the taps of other zones. The taps can be positioned along paths, and paths of the power taps can be spaced from associated proximate paths of supply and return taps by a substantially uniform distance along a substantial length of the first path.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.12/146,281, filed Jun. 25, 2008, which claims priority to U.S.Provisional Application Ser. No. 60/946,699, filed Jun. 27, 2007, theentire disclosure of which is incorporated by reference.

TECHNICAL FIELD

This document relates to systems and methods for providing cooling andpower to an area containing electronic equipment, such as a computerdata center containing server racks.

BACKGROUND

Higher speed computers come with a cost—higher electrical consumption.For a small number of home PCs this extra power may be negligible whencompared to the cost of running other electrical appliances in ahousehold. However, in data center applications, where thousands or tensof thousands of microprocessors may be operated, electrical powerconsumption becomes important.

In addition, the power consumed by a microprocessor is transformed intoheat. A pair of microprocessors mounted on a single motherboard can draw200-400 watts or more of power. If that power draw is multiplied byseveral thousand (or tens of thousands) to account for the computers ina data center, the potential for heat generation can be appreciated.Thus, not only must a data center operator pay for electricity tooperate the computers, it must also pay to cool the computers. The costof removing heat may be a major cost of operating large data centers.

A typical approach to removing heat in a data center uses airconditioning, e.g., cold air is blown through the room containing thecomputers. For example, a current common practice is to construct a datacenter on a raised floor, and use a computer room air conditioner toforce cold air through ducts below the floor and up through holes in thefloor beneath or between the server racks. The cold air flows over themicroprocessors and is heated, and the heated air can be drawn throughceiling ducts back to the computer room air conditioner.

SUMMARY

In one aspect, a facility is described that includes one or moreenclosures defining an interior space, a plurality of power tapsdistributed with a substantially regular spacing in the interior space,a plurality of coolant supply taps distributed with a substantiallyregular spacing in the interior space, and a plurality of coolant returntaps distributed with a substantially regular spacing in the interiorspace. A flow capacity of the supply taps and a flow capacity of thereturn taps are approximately equal over a local area of the interiorspace.

Implementations of the invention may include one or more of thefollowing features. The supply taps and return taps may be connected toa heat exchanger or cooling plant. There need not be local storage orbuffering of the coolant in the interior space. The plurality of powertaps may be distributed with a substantially regular first spacing in afirst direction and with a substantially regular second spacing in asecond direction perpendicular to the first direction, the plurality ofsupply taps may be distributed with a substantially regular thirdspacing in the first direction and with a substantially regular fourthspacing in the second direction, and the plurality of return taps may bedistributed with a substantially regular fifth spacing in the firstdirection and with a substantially regular sixth spacing in the seconddirection. The third spacing may be approximately equal to the fifthspacing and the fourth spacing is approximately equal to the sixthspacing. The first spacing may be a ratio N/M of the third spacing,where N and M are both whole numbers less than 5. The second spacing maybe an integer N multiple (N) or fraction (1/N) of the fourth spacing,where N is less than five. The first spacing may be approximately equalto the third spacing and the second spacing may be approximately equalto the fourth spacing. The plurality of power taps may be laid along aplurality of first paths, the supply taps may be laid along a pluralityof second paths, and the return taps may be laid along a plurality ofthird paths. The power taps may be disposed with substantially uniformspacing along the first paths, the supply taps may be disposed withsubstantially uniform spacing along the second paths, and the returntaps may be disposed with substantially uniform spacing along the thirdpaths. A spacing of the power taps along the first paths may be lessthan a spacing between adjacent first paths, a spacing of the supplytaps along the second paths may be less than a spacing between adjacentsecond paths, and a spacing of the return taps along the third paths maybe less than spacing between adjacent third paths. The first paths,second paths and third paths may be substantially are substantiallylinear. A plurality of power delivery busbars may provide power to theplurality of power taps, and the busbars may define the first paths. Aplurality of coolant supply manifolds may provide coolant to theplurality of supply taps, and the coolant supply manifolds may definethe second paths. A plurality of coolant return manifolds may returncoolant from the return taps, and the coolant return manifolds maydefine the third paths. Each first path of the plurality of first pathsmay have an associated proximate second path of the plurality of secondpaths and an associated proximate third path from the plurality of thirdpaths, and each first path may be spaced from the associated proximatesecond path by a substantially uniform first distance along asubstantial length of the first path and each first path may be spacedfrom the associated proximate third path by a substantially uniformsecond distance along a substantial length of the first path. There maybe a plurality of zones, each zone including a different set of two ormore power taps from the plurality of power taps, a different set of twoor more supply taps from the plurality of supply taps, and a differentset of two or more return taps from the plurality of return taps. Thetwo or more power taps of each zone may be configured to be controllablyelectrically coupled to a power source independently of the power tapsof other zones. The two or more supply taps of each zone may beconfigured to be controllably fluidly coupled to a coolant sourceindependently of the supply taps of other zones, and the two or morereturn taps of each zone may be configured to be controllably fluidlycoupled to a coolant return independently of the return taps of otherzones. The power taps may be spaced at a substantially uniform firstdensity across the interior space, the coolant supply taps may be spacedat a substantially uniform second density across the interior space, andthe coolant return taps may be spaced at spaced at a substantiallyuniform third density across the interior space. The second density maybe approximately equal to the third density. Each supply tap may includea spigot, and each spigot may include a valve and a faucet. Each powertap may include a plurality of outlets.

In another aspect, a facility is described that includes one or moreenclosures defining an interior space, a plurality of power tapsdistributed with a substantially regular spacing in the interior space,a plurality of coolant supply taps distributed with a substantiallyregular spacing in the interior space, and a plurality of coolant returntaps distributed with a substantially regular spacing in the interiorspace. The plurality of power taps, the plurality of supply taps, andthe plurality of return taps are divided into a plurality of zones, eachzone including a different set of two or more power taps from theplurality of power taps, a different set of two or more supply taps fromthe plurality of supply taps, and a different set of two or more returntaps from the plurality of return taps. The two or more power taps ofeach zone are configured to be controllably electrically coupled to apower source independently of the power taps of other zones and the twoor more supply taps of each zone are configured to be controllablyfluidly coupled to a coolant source independently of the supply taps ofother zones, and wherein the two or more return taps of each zone areconfigured to be controllably fluidly coupled to a coolant returnindependently of the return taps of other zones.

Each zone may be a spatially contiguous area separate from other zones.

In another aspect, a facility is described that includes one or moreenclosures defining an interior space, a plurality of power tapspositioned along a plurality of first paths in the interior space, aplurality of coolant supply taps positioned along a plurality of secondpaths in the interior space, and a plurality of coolant return tapspositioned along a plurality of third paths in the interior space. Eachfirst path of the plurality of first paths has an associated proximatesecond path of the plurality of second paths and an associated proximatethird path from the plurality of third paths, and wherein each firstpath is spaced from the associated proximate second path by asubstantially uniform first distance along a substantial length of thefirst path and wherein each first path is spaced from the associatedproximate third path by a substantially uniform second distance along asubstantial length of the first path.

Implementations of the invention may include one or more of thefollowing features. The power taps may be distributed with asubstantially regular spacing, the supply taps may be distributed with asubstantially regular spacing, and the return taps may be distributedwith a substantially regular spacing. The supply taps and return tapsmay have the same spacing. A plurality of data taps may be distributedwith a substantially regular spacing in the interior space. A pluralityof data taps may be positioned along a plurality of fourth paths in theinterior space, each first path of the plurality of first paths may havean associated proximate fourth path of the plurality of second paths,and each first path may be spaced from the associated proximate fourthpath by a substantially uniform fourth distance along a substantiallength of the first path. The power taps may be disposed withsubstantially uniform first spacing along the first paths, the supplytaps may be disposed with substantially uniform second spacing along thesecond paths, and the return taps may be disposed with substantiallyuniform third spacing along the third paths. A spacing of the power tapsalong the first paths may be less than a spacing between adjacent firstpaths, a spacing of the supply taps along the second paths may be lessthan a spacing between adjacent second paths, and a spacing of thereturn taps along the third paths may be less than spacing betweenadjacent third paths. The first paths, second paths and third paths maybe substantially linear. The second paths and the third paths may beuniformly spaced with a first pitch. The second paths may be immediatelyadjacent to the third paths. Adjacent second paths and third paths maybe separated by one-half of the first pitch. The first paths may beuniformly spaced with a second pitch. The second pitch may be an integerN multiple (N) or fraction (1/N) of the first pitch, where N is lessthan five. The second pitch may be equal to the first pitch. The secondpaths and third paths may be arranged substantially parallel to thefirst paths. The first spacing may be a ratio N/M of the second spacing,where N and M are both whole numbers less than 5. The second spacing maybe approximately equal to the third spacing. A plurality of powerdelivery busbars may provide power to the plurality of power taps andthe busbars may define the first paths, a plurality of coolant supplymanifolds may provide coolant to the plurality of supply taps and thecoolant supply manifolds may define the second paths, and a plurality ofcoolant return manifolds may return coolant from the return taps and thecoolant return manifolds may define the third paths. Each supply tap mayinclude a spigot, and each spigot may include a valve and a faucet. Eachpower tap may include a plurality of power outlets. A flow capacity ofthe supply taps and a flow capacity of the return taps may beapproximately equal over a local area of the interior space. There maybe no local storage or buffering of the coolant. The plurality of supplytaps and the plurality of return taps may be connected to a heatexchanger or cooling plant. A plurality of cooling coils may remove heatfrom air near the rack-mounted computers, and the cooling coils may befluidly connected between the supply taps and the return taps.

Each facility may have an associated method of building the facilityincluding building one or more enclosures, placing power taps, placingcoolant supply taps and placing coolant return taps.

In another aspect, a data center is described. The data center includesone or more enclosures defining an interior space, a plurality of powertaps distributed with a substantially regular spacing in the interiorspace, a plurality of coolant supply taps distributed with asubstantially regular spacing in the interior space, a plurality ofcoolant return taps distributed with a substantially regular spacing inthe interior space, and a plurality of modules. Each module includes aplurality of rack-mounted computers connected to a power tap adjacentthe module and a cooling coil to remove heat from air near therack-mounted computers, the cooling coil fluidly connected between asupply tap and a return tap adjacent the module.

Implementations can include one or more of the following. A plurality ofpower lines may have the plurality of power taps, a plurality of coolantsupply lines may have the plurality of coolant supply taps, and aplurality of coolant return lines may have the plurality of coolantreturn taps. The power lines, coolant supply lines and coolant returnlines may be substantially linear. The modules may be arranged insubstantially linear rows. The linear rows of module may beperpendicular or parallel to the power lines. There may be a power linefor each module in a row of the modules or for each row of modules. Thelinear rows of modules may be perpendicular or parallel to the coolantsupply lines and the coolant return lines. There may be a coolant supplyline and a coolant return line for each row or every two rows ofmodules. The power lines may be positioned above of the modules, and thecoolant supply lines and the coolant return lines may be positionedbelow the modules. Each module may be connected to a coolant supply tapon one side of the module and to a coolant return tap on an oppositeside of the module, or each module may be connected to a coolant supplytap and to a coolant return tap on the same side of the module. A flowcapacity of the supply taps and a flow capacity of the return taps maybe approximately equal over a local area of the interior space. Theplurality of power taps may be distributed with a substantially regularfirst spacing in a first direction and with a substantially regularsecond spacing in a second direction perpendicular to the firstdirection, the plurality of supply taps may be distributed with asubstantially regular third spacing in the first direction and with asubstantially regular fourth spacing in the second direction, and theplurality of return taps may be distributed with a substantially regularfifth spacing in the first direction and with a substantially regularsixth spacing in the second direction. The third spacing may beapproximately equal to the fifth spacing and the fourth spacing may beapproximately equal to the sixth spacing.

In another aspect, a data center is described. The data center includesone or more enclosures defining an interior space, a plurality of powerlines, a plurality of coolant supply lines, a plurality of coolantreturn lines, and a plurality of clusters of modules in the interiorspace. Each of the power lines includes a plurality of power taps in theinterior space, each of the coolant supply lines includes a plurality ofcoolant supply taps in the interior space, and each of the coolantreturn lines includes a plurality of coolant return taps in the interiorspace. Each cluster is located in a spatially contiguous area separatefrom other clusters, each module includes a plurality of rack-mountedcomputers connected to a power tap adjacent the module and a coolingcoil to remove heat from air near the rack-mounted computers, thecooling coil fluidly connected between a supply tap and a return tapadjacent the module. Each cluster includes two or more modules, and eachof the two or more modules is connected to different ones of theplurality of power lines or different ones of the plurality of coolantsupply lines and the plurality of coolant return lines.

Implementations may include one or more of the following. Each of thetwo or more modules may be connected to different ones of the pluralityof power lines. Each of the two or more modules may be connected todifferent ones of the plurality of coolant supply lines and theplurality of coolant return lines. Each of the two or more modules maybe connected to different ones of the plurality of power lines anddifferent ones of the plurality of coolant supply lines and theplurality of coolant return lines. Substantially all of the rack-mountedcomputers of a particular cluster may be dedicated to the sameapplication. The rack-mounted computers of at least two differentclusters of the plurality of clusters may be dedicated to differentapplications. The plurality of power taps may be distributed with asubstantially regular spacing in the interior space, the plurality ofcoolant supply taps may be distributed with a substantially regularspacing in the interior space, and the plurality of coolant return tapsmay be distributed with a substantially regular spacing in the interiorspace. The power lines, coolant supply lines and coolant return linesmay be substantially linear. The modules may be arranged insubstantially linear rows. The linear rows of the modules may beperpendicular to the power lines. The linear rows of the modules may beparallel to the coolant supply lines and the coolant return lines. Thepower lines may be uniformly spaced. The power lines may be spaced witha pitch approximately equal to a spacing between modules in the rows.The linear rows of the modules may be perpendicular to the coolantsupply lines and the coolant return lines. The linear rows of themodules may be parallel to the power lines. The linear rows of themodules may be perpendicular to the power lines, the coolant supplylines and the coolant return lines. The rows of modules may be separatedby access aisles. At least two of the coolant supply lines and at leasttwo of the coolant return lines may be connected to different coolingplants. The power taps may be disposed with substantially uniform firstspacing along the power lines, the supply taps may be disposed withsubstantially uniform second spacing along the coolant supply lines, andthe return taps may be disposed with substantially uniform third spacingalong the coolant return lines. The second spacing may be approximatelyequal to the third spacing. The coolant supply lines and the coolantreturn lines may be uniformly spaced with a first pitch. The modules maybe distributed with a substantially regular spacing in the interiorspace.

In another aspect, a method of operating a data center is described. Themethod includes supplying power to rack-mounted computers in a row ofracks from a plurality of power taps extending along the row, supplyingcoolant to coolant coils in a space adjacent the racks from a pluralityof supply taps of a coolant supply manifold extending along the row, anddirecting warmed coolant from the coolant coils through a plurality ofreturn taps of a coolant return manifold extending along the row.

Implementations of the invention may include one or more of thefollowing. Heat may be removed from the warmed coolant and returning thecoolant to the coolant supply manifold. The servers may be located in aninterior space, the power taps may be spaced at a substantially uniformfirst density across the interior space, the supply taps may be spacedat a substantially uniform second density across the interior space, andthe return taps may be spaced at spaced at a substantially uniform thirddensity across the interior space.

The details of one or more embodiments of the invention are set forth inthe accompanying drawings and the description below. Other features,objects, and advantages of the invention will be apparent from thedescription and drawings, and from the claims.

DESCRIPTION OF DRAWINGS

FIGS. 1A and 1B are sectional and plan views, respectively, of afacility with cooling and power grids.

FIGS. 2A and 2B are sectional and plan views of the facility of FIGS.1A-1B operating as a data center.

FIGS. 3A and 3B are sectional and top views of an exemplary computingmodule.

FIGS. 4-13 are plan views of other implementations of a facility withcooling and power grids operating as a data center.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

One issue with using computer room air conditioners for cooling a datacenter is efficiency. Since air has a low heat capacity, it would bemore efficient to place the coolant (e.g., water) closer to themicroprocessors. One example of such a technique is to place coolingcoils on the sides of a server rack or group of server racks. Such atechnique is described by U.S. Patent Application Ser. No. 60/810,452,filed Jun. 1, 2006, the entire disclosure of which is incorporated byreference.

Another issue with any data center is time in bringing a data centerinto operation; the faster a data center can begin operating the soonerit can generate revenue. Moreover, if a portion of a data center, e.g.,one or more server racks, could be brought into operation while otherserver racks in the data center are still being installed, this wouldprovide significant flexibility in construction and operation whilepermitting the portion that is operating to meet data processing needs.This issue is exasperated with computer room air conditioners; not onlyis a facility generally not suitable for installation of server racksuntil the entire air conditioning system is installed, but if only aportion of the data center is operating then the air conditioning unitmust cool the entire room rather than just the operating portion.

FIG. 1A shows a side sectional view of a facility 10 with a power grid30, a cooling grid 50 and a “data grid” 70, before installation of thecomputers and other components of the data center. FIG. 1B shows a planview of the facility 10 with the power grid 30, cooling grid 50 and“data grid” 70. The facility 10 is an enclosed space and can occupyessentially an entire building, or be one or more rooms within abuilding. The facility 10 can include a floor 12 (e.g., a concrete slabor a metal sheeting laid over steel beams), a raised floor 14 supportedon the floor 12, a ceiling 16 (which could be the floor of another levelof the building) and a suspended ceiling 18 hung from support beams orthe ceiling 16. Doors 19 can be formed in the walls of the facility.Between the raised floor 14 and the suspended ceiling 18 is an enclosedspace 20 sufficiently large for installation of numerous (dozens orhundreds or thousands of) server racks.

As shown in FIGS. 1A and 1B, the power grid 30 includes a distributedset of power “taps” 34, e.g., outlets or receptacles, e.g., sockets. Thepower taps can be distributed in a line or array of regularly spacedpower taps. In operation, the power grid 30 is connected to a powersupply, e.g., a generator or an electric utility, and suppliesconventional commercial AC electrical power, e.g., 120 or 208 Volt, 60Hz (for the United States). The receptacles can be conventionalcommercial power recepticles, e.g., Type B (American 3-pin) (again, forthe United States). The outlets can be flush or recessed in thesuspended ceiling 18, or the outlets can hang below the suspendedceiling 18. The amperage capacity of the busbar 32 can be selectedduring installation to exceed the expected draw, which can be calculatedbased on the maximum number of server cabinets that could draw powerfrom that busbar.

As shown in FIG. 1B, the power grid can include power distribution“lines” 32, such as busbars 32 suspended on or from the ceiling 18. Eachbusbar 32 includes the power taps 34, e.g., the outlets or receptacles.Alternatively, busbars could be replaced by groups of outletsindependently wired back to the power supply, e.g., elongated plugstrips or receptacles connected to the power supply by electrical whips.Optionally, a group of taps, e.g., the taps along a particular line 32,can be connected by a common switch or circuit breaker to the powersupply so that power can be shut off to the line of taps as a group.

The power taps 34 have a regular spacing, e.g., a regularly repeatingpattern of spacing, such as at regular intervals with a spacing D1,along the busbar 32. Each power tap 34 can include a cluster ofelectrical outlets, e.g., six to twelve outlets, held in an singleframe. The power tap 34 can include more than twelve outlets, or as fewas a single outlet.

As shown in FIG. 1B, the power grid 30 can include multiple power supplylines, e.g., busbars 32, spaced evenly across the facility area with apitch P1. Thus, the power “taps” 34 can be distributed uniformly acrossthe space of the facility. The busbars 32 can be connected to a commonelectrical supply line 36, which in turn can be connected to the powersupply.

FIGS. 2A and 2B show sectional and plan views of the facility 10 withseveral modules 100 of rack-mounted computers installed. The modules 100are arranged in rows 102 separated by access aisles 104. Each module 100can include multiple racks, and each rack can include multiple trays.The busbars 32 can extend parallel to the row of modules.

As shown, each module 100 is connected to an adjacent power tap 34,e.g., by power cabling 38. Assuming that the busbars 32 do run parallelto a row of modules, the spacing D1 can be equal to or less than thewidth of a module that is expected to be installed. For example, thespacing D1 can be about two to six feet. Alternatively, the spacing canbe greater than the width W of the module (although probably not morethan two or three times the width of a module), in which case the numberof outlets can be increased proportionally. The pitch P1 can be aboutequal to the pitch between the rows of modules, e.g., six to twelvefeet.

Although FIGS. 2A and 2B illustrate the rows of modules 100 as adjacentthe busbars 32, the rows of modules could be directly beneath thebusbars. In addition, the busbars 32 need not be suspended from theceiling, but can run through the plenum 22 between the floor 12 and theraised floor 14.

Returning to FIG. 1A, the cooling grid 50 includes a coolant, e.g.,water, supply manifold 52 and a coolant return manifold 62 which runthrough the plenum 22 between the floor 12 and the raised floor 14.Although this specification describes water supply and return manifolds,other cooling fluids or refrigerants could be used. The water supplymanifold 52 includes supply “taps” 54, e.g., water spigots, each havinga valve 56 and an outlet 58 that is threaded or otherwise configured,e.g., with a threadless quick-disconnect type fitting, for fastening ahose or pipe. The water supply taps 54 have a regular spacing, e.g., aregularly repeating pattern of spacing, such as at regular intervalswith a distance D2, along the supply manifold 52. Similarly, the waterreturn manifold 62 includes return “taps” 64, each having a valve 66 andan inlet 68 that is threaded or otherwise configured for fastening ahose or pipe (these water inlets can be constructed similarly toconventional spigots, but are for water return rather than supply). Thewater return taps have a regular spacing, e.g., a regularly repeatingpattern of spacing, such as at regular intervals with a distance D3,along the return manifold 62. Each supply “tap” 54 can include a singlespigot or a cluster of spigots, and similarly each return tap 64 caninclude a single inlet or a cluster of inlets. The spigots and inletscan project above the raised floor 14, or be flush with or recessed inthe floor 14. In some implementations, the spacing D2 of the watersupply taps 54 is the same as the spacing D3 of the water return taps64.

The flow capacity of the supply and return manifolds 52/62 can beselected during installation to exceed the expected draw, which can becalculated based on the maximum power available for server cabinets thatcould draw water from or return water to that manifold. The manifold canbe 4-inch or 6-inch diameter piping, e.g., PVC or steel pipe.

The water supply and return manifolds 52 and 62 are connected by a pump40 and a heat exchanger or cooling plant 42, which can be locatedoutside the space that actually holds the rack-mounted computers butwithin or adjacent the same building. The flow capacity of the supplyand return manifolds 52 and 62, the flow capacity of the pump 40 and thecapacity of the heat exchanger 42 can be selected before installationfor the expected heat based on the power available in the associatedbusbars.

As shown in FIG. 1B, the cooling grid 50 can include multiple watersupply and return manifolds 52 and 62, spaced evenly across the facilityarea with a pitch P2 and P3 respectively. Thus, the water supply andreturn “taps” 54 and 64 can be distributed uniformly across the space ofthe facility. The supply manifolds 52 can be connected to a common watersupply line 59, and the return manifolds 62 can be connected to a commonwater return line 69, and the water supply line 59 and water return line69 can in turn can be connected to the pump 40 and a heat exchanger orcooling plant 42. A valve can couple each supply manifold 52 and returnmanifold 62 to the water supply line 59 and water return line 69 so thatcoolant flow can be shut off to a line of taps (and the associated lineof server racks) as a group.

In some implementations, the pitch P2 of the water supply manifolds 52can be the same as the pitch P3 of the water return manifolds 62. Insome implementations, the density of the water return taps 64 (e.g., perunit length of the water return manifold) times the flow capacity perwater return tap can be equal to the density of the water supply taps 54(e.g., per unit length of the water supply manifold) times the flowcapacity per water supply tap. The flow capacity per water tap can beadjusted by modulating valves, selecting the number spigots or inletsper tap. In general, the supply and return flow is matched on a localbasis (e.g., over several modules) so that there is no local storage orbuffering of the water. In some implementations, each water supply tap54 can be located adjacent a water return tap 64.

Although FIGS. 2A and 2B illustrate the rows of modules 100 as adjacentthe water supply and water return manifolds 52 and 62, the rows ofmodules 100 could be directly above the manifolds. In addition, themanifolds need not run through the plenum 22, but could be suspendedfrom the ceiling.

As shown, each module 100 is connected to an adjacent water supply tap54 and water return tap 64, e.g., by flexible hoses 44 and 46,respectively. Assuming that the manifolds run parallel to a row ofmodules, the spacing D2 and D3 can be equal to or less than the width ofa module that is expected to be installed. For example, the spacings D2and D3 can be about four to six feet. If the spacing D2 or D3 is greaterthan the width of the modules, then the number of spigots or inletswould need to be increased. The pitch P2 and P3 can be about equal tothe pitch between the lines of server racks, e.g., four to twelve feet.

Generally, the spacing D1 is related to the spacing D2 and D3, and thepitch P1 is related to the pitch P2 and P3. In a simple implementationas shown in FIGS. 1A-2B, the pitches P2 and P3 of the manifolds of thecooling grid are equal the pitch P1 of the busbars of the power grid. Inaddition, the spacings D2 and D3 of the cooling grid taps are equal tothe spacing D1 of the power taps. For example, the cooling grid taps canbe spaced 2 feet apart, whereas the power grid taps can have a spacingof 600 millimeters.

Returning to FIG. 1A, the “data grid” 70 (the data grid is not shown inFIG. 1B due to lack of space) includes data cabling 72 and data “taps”74, e.g., data outlets, for connection to the rack-mounted computers.The cabling can be conventional Cat-5 or Cat-6 or CX-4 or opticalcabling, and the data outlets can be conventional modular receptacles.Each data tap 74 can include a cluster of data outlets, e.g., 180outlets. The data taps 74 have a regular spacing, e.g., a regularlyrepeating pattern of spacing, such as at regular intervals with adistance D4, along the string of cabling 72. The data cabling 72 can besuspended from the ceiling as shown or run through the plenum 22 betweenthe floor 12 and the raised floor 14. Alternatively, data cabling couldrun directly to the rack-mounted computers without an interveningreceptacle. Groups of cables, e.g., the data cables from a rack ofrack-mounted computers, a row of racks, or a line of computer modules,can be connected to an intervening switch or patch board.

The data grid 70 can include multiple cabling bundles 72, spaced evenlyacross the facility area with a pitch P4 (not shown). Thus, the data“taps” 74 can be distributed uniformly across the space of the facility.

As shown in FIG. 2A, each rack-mount module 100 is connected to anadjacent data tap 74 by data cabling 78, e.g., by additional Cat-5 orCat-6 cabling. The data cabling 72 can extend parallel to the row ofmodules. Assuming that the data cables 72 run parallel to a row ofmodules, the spacing D4 can be equal to or less than the width of amodule that is expected to be installed. For example, the spacing D4 canbe about four to six feet. Alternatively, the spacing can be greaterthan the width of the module, in which case the number of datareceptacles can be increased proportionally.

The spacing D4 can be related to the spacing D1, D2 and D3, and thepitch P4 can be related to the pitch P1, P2 and P3. In a simpleimplementation as shown in FIGS. 1A-2B, the pitches P4 of the datacabling 72 is equal to the pitch P1 of the busbars of the power grid andthe spacing D4 of the data taps is equal to the spacing D1 of the powertaps.

In general, the rows of modules 100 extend along the same path, e.g.,are linear and parallel to, the busbars 32, water supply manifolds 52and water return manifolds 62. In the implementation illustrated inFIGS. 1A-2B, there is one row of modules 100 per busbar 32, water supplymanifold 52 and water return manifold 62. However, as discussed below,many other layouts are possible.

In some implementations, the rack-mounted computers are grouped (e.g., agroup can be the computers in the modules along a particular path, suchas a row of modules), and the power and coolant flow can beindependently shut on and off for each group, e.g., power and coolantcan be shut off for one row of modules so that the computers in thatgroup are disabled, while power and coolant continue to flow and thecomputers continue to operate in another row of modules. However, insome layouts, it may be possible to shut off coolant to a group ofmodules (e.g., multiple rows), whereas power can be shut offindependently to sub-groups within the group (e.g., individual rows).

FIGS. 3A and 3B are side and top views of a rack-mount computer module100 that can be installed in the facility. The module 100 includes twoparallel rows 110 of racks 120 separated by a space 112. Each row 110 inthe module 100 can include several, for example, two to five, e.g.,three, racks 120 placed side-by side. Each rack 120 can include severaldozen vertically stacked trays 122, with approximately several inchesbetween each tray. The term “tray” is not intended to refer to aparticular form factor, but refers to any arrangement ofcomputer-related components coupled together so as to be removable fromthe rack as a unit. Moreover, the term “computer module” or“rack-mounted computer” includes not just rack-mounted computers, e.g.,servers, but also racks of other communications and data processingequipment, such as network gear, e.g., switches and routers.

In general, each tray 122 can include a circuit board, such as amotherboard, on which a variety of computer-related components aremounted. Trays can be implemented for particular functional purposes,such as computer servers (whether for electronic mail, search requests,or other purposes), network gear (such as switches or routers), datastorage (with a drive or group of drives). A given rack can includetrays dedicated to a single function, or a rack can include a mix oftrays with different functions. In general, trays in a data center havea standardized physical and power-coupling form so as to be easilyinterchangeable from one location in the data center to another (e.g.,from one slot on a rack to another slot or from one rack to anotherrack). Trays for a given functional purpose can also have a standardizedform for the physical layer of their input/output interface. Foroperation, each circuit board will be connected both to the power grid,e.g., by wiring that first runs through the rack itself and which isfurther connected by power cabling to a nearby power tap 34, and to thedata grid, e.g., by data cabling that is connected to a nearby data tap74.

Cooling coils 130 are located in the space 112 between the rows 110 ofracks 120. For operation, one end of the cooling coil 130 is connected,e.g., by a flexible hose 44, to a nearby water supply tap 54 and theother end of the cooling coil 130 is connected, e.g., by a flexible hose46, to a nearby water return tap 56. In addition, fans 132 can be placedon the walls of the racks and above the space and be powered by thebusbar.

In operation, cool water will be pumped through the supply manifolds andinto the cooling coil 130 via the taps 54. The fans 132 will draw airacross the trays 122 into the space 112, thereby removing heat from themicroprocessors in the trays. This warmed air is drawn through thecooling coils 130 and directed up through the top of the space 112. Thecooling coils 130 transfer heat from the air passing through the space112 to the water in the coils, and the warmed water is drawn through thereturn taps 56 and return manifolds back to the heat exchanger orcooling plant 42.

FIG. 4 is a plan view of an implementation of a facility with a coolingand power grids and operating as a data center with two rows of modules100 per busbar 32, water supply manifold 52 and water return manifold62. Each module 100 is connected to an immediately adjacent grid. Thus,at least some of the busbars 32, water supply manifolds 52 and waterreturn manifolds 62 are connected to rows of modules 100 on oppositesides. This permits the water supply and return manifolds 52 and 62 tobe placed beneath the flooring of the aisles 104 separating the modules100 where the manifolds can be more easily serviced. Although some ofthe modules are illustrated as adjacent, if the modules have racks onopposite sides, then the modules would need to be spaced apart toprovide an aisle for personnel access.

The implementation shown in FIG. 5 is similar to FIG. 4, but a powerbusbar 32 is disposed over each row of modules 100. Thus, in thislayout, the pitches P2 and P3 are two times greater than pitch P1 andare offset from pitch P1 by about P1/2.

FIG. 6 is a plan view of another implementation of a facility withcooling and power grids and operating as a data center. As shown, thespacings D1, D2 and D3 are equal and the pitches P1, P2 and P3 areequal, but the power busbars 32 are spaced a half-pitch P1 or P2 fromthe water supply and return manifolds 52 and 62. In addition, eachmodule 100 can be connected to a supply manifold 52 and a returnmanifold 62 that are located on opposite sides of the module. Thus, inthis layout, the supply manifold 52 and return manifolds are offset bysubstantially an entire pitch P1 or P2. This configuration permits thebusbars 32 to run directly over the modules 100, but water supply andreturn manifolds 52 and 62 can be placed beneath the flooring of theaisles 104 separating the modules 100 where the manifolds can be moreeasily serviced. In addition, if any manifold has to be shut down, thenonly the single row of computers components serviced by that manifoldwould also need to be shut down.

FIG. 7 is a plan view of another implementation of a facility withcooling and power grids and operating as a data center. As shown, ratherthan having immediately adjacent water supply and return manifolds 52and 62, each water supply manifold 52 is separated from the nearestwater return manifold 62 by a row of modules 100. Thus, the pitches P2and P3 are two times greater than pitch P1. The spacings D1, D2 and D3can be equal, but the minimum number of spigots and inlets at each tap(excepting those along the edges of the room) would be twice the minimumnumber for the implementations shown in FIG. 5. This reduces the amountof installed pipe and thus can reduce construction cost and time, but ifany manifold has to be shut down then two rows of modules will be alsoneed to be shut down.

FIG. 8 is a plan view of another implementation of a facility with acooling and power grids operating as a data center. In thisimplementation, each pair of water supply and return manifolds 52 and 62is separated by multiple rows, e.g., four or fewer rows, of modules 100.Thus, the pitches P2 and P3 are two or more times greater than pitch P1(only one supply manifold is shown in FIG. 8 due to space limitations,and thus pitch P2 is not illustrated). The spacings D1, D2 and D3 can beequal, but the minimum number of spigots and inlets at each tap would betwo or more the minimum number for the implementations shown in FIG. 7.Although this uses longer hoses than the implementation shown in FIG. 7,it reduces the amount of installed pipe and thus reduces constructioncost and time. Of course, the features of FIGS. 4 and 5 could becombined with each water supply manifold 52 separated from the nearestwater return manifold 62 by multiple rows of modules 100. In addition toor instead of the water supply and return manifolds being separated bymultiple rows of modules, the electrical busbars 32 could be separatedby multiple rows, e.g., four or fewer rows, of modules 100.

As shown in FIG. 9, the water supply and return taps can be arranged ina simple repeating pattern (e.g., no more than two or three differentdistances). For example, the distance between adjacent taps 54 canalternate between first distance 51 and second distance S2. The averagedistance (S1+S2)/2 can be two to ten feet. For example, the firstdistance 51 can be four feet, and the second distance be six feet. Thedistances 51 and S2 can be a simple ratio, e.g., S1=(N/M)*S2 where N andM are both less than 5. N/M can be less than 3. This permits tap spacingto be coordinated with raised floor tile/stanchion pitch (2 ft in US)while accommodating module 100 spacings that are not integer multiplesof 2 ft.

As shown in FIG. 10, as a more general case, rather than having thepower taps 34 and water taps 54/64 have the same spacing, the distancesD1 and D2 can be a ratio of small integers, e.g., D1=(N/M)*D2 where Nand M are both less than 5. Similarly, the distances D4 between adjacentdata taps can be a simple ratio of the distances D1 or D2 and D3. Havingthe distances be a ratio of small integers can simplify planning of datacenter layout. However, other ratios are possible. The power taps can bemore closely spaced than the water taps 54/64.

The spacing of the coolant taps 54/64 (and number of spigots per tap)can be selected such that when a full complement of modules 100 has beeninstalled and connected to an associated cooling line, substantiallyall, e.g., more than 90%, of the coolant taps on the line are used.

In general, since power taps are less costly than coolant taps, andbecause power taps may be needed for other purposes, the data center canbe installed with more power taps than would otherwise be required bythe modules. This also permits greater flexibility in electricalconnection to the modules 100. The spacing of the power taps 34 (andnumber of receptacles per tap) can be selected such that when a fullcomplement of modules 100 has been installed and connected to anassociated power line, the modules use a plurality or majority butsignificantly less than all of the taps. For example, the modules canuse more than 30%, e.g., more than 40%, e.g., more than 50%, of thepower taps on the line. On the other hand, the modules can use less than90%, e.g., less than 80%, e.g., less than 70%, of the power taps on theline. In one implementation, the modules use about 50% of the power tapson the line.

In addition, each pair of adjacent busbars or manifolds need not havethe same spacing, but can be arranged in a simple repeating pattern(e.g., no more than two or three different distances). For example, thespacing between busbars can be closer for two busbars over modules thatare disposed back-to-back (e.g., as shown in FIG. 5) than for twobusbars over modules that are disposed across an aisle.

In general, the density of power taps 34, water taps 54/64 and data taps74 can be such that no hose or cabling need be longer than about tenfeet to connect the components of a module to the nearest taps. Thedensity of power taps 34, water taps 54/64 and data taps 74 can each besubstantially uniform across the area of the facility to be used for themodules (non-uniformity is more likely to arise along the walls of thebuilding). Assuming that the ratio of water supply manifolds to powersupply busbars is greater than 1:1, then each power supply busbar 32 canhave a limited number of associated water supply and return manifolds(e.g., four or less each) that serve the same group of modules.Similarly, assuming that the ratio water supply manifolds to powersupply busbars is less than 1:1, then each pair of water supply andreturn manifolds can have a limited number of associated power supplybusbars 32 (e.g., four or less each) that serve the same group ofmodules.

FIG. 11 is a plan view of another implementation of a facility withcooling and power grids operating as a data center. As shown, the linesof power taps 34 and water taps 54/64 can be arranged perpendicular tothe rows 102 of modules 100. For example, the coolant supply manifolds52 and coolant return manifolds 62 can extend under the raised flooringperpendicular to the rows 102 of modules 100 with taps 54 and 64 placedin the aisles 104 between the rows 102. The busbars 32 can runperpendicular to the rows 102, with taps 34 over the modules 100. Ingeneral, in this implementation, the spacing of the taps along the linescan be about equal to the pitch between the rows of modules, and thepitch of the lines of taps can be about equal to the spacing of moduleswithin a row. However, the various modifications to tap layout describedabove with respect to FIGS. 4-10, e.g., modifications to pitch, spacing,staggering of the taps and lines, etc., can also be applied to theperpendicular arrangement. For example, coolant taps can be locatedevery other aisle and service the modules on both sides of the aisle, orthe supply and return taps for a given module can be located in aisleson opposite sides of a module.

A potential advantage to running the lines perpendicular to the row ofmodules is distribution of cooling and power load across differentapplications. Assuming that the rack-mounted computers in a given row102 of modules 100 have similar applications, but that different rows ofmodules have different applications, then the perpendicular arrangementallows rack-mounted computers for different applications to be cooled bya common cooling plant. This can create an averaging effect on thecooling plant loads, so that spikes in activity for particularapplications can be less likely to overwhelm the cooling capacity of aparticular cooling plant. Similarly, the perpendicular arrangementallows rack-mounted computers for different applications to be poweredby a common power supply. This can create an averaging effect on thepower loads, so that spikes in activity for particular applications canbe less likely to overwhelm the power capacity of a particular upstreampiece of power distribution infrastructure, e.g., the capacity of aparticular power distribution unit (PDU). This averaging effect andreduction in the spikes, can allow the deployment of an increased numberof servers while still maintaining the appropriate equipment safetymargins.

Although FIG. 11 illustrates both the lines of power taps 34 and watertaps 54/64 arranged perpendicular to the rows 102 of modules 100, inother implementations only the lines 21 of power taps 34 (see FIG. 12)or only the lines 52/62 of water taps 54/64 (see FIG. 12) are arrangedperpendicular to the rows 102 of modules 100. In particular, in oneimplementation, the lines 32 of power taps 34 are perpendicular to therows 102 of modules and the lines of water taps 54/64 are parallel tothe rows 102 of modules 100. It can be more advantageous to provisionthe power perpendicularly than the cooling because insufficient coolingis a soft failure (a slight to modest increase in temperature) while forpower provisioning, the failure mode is the tripping of a breaker orburning of a fuse. Thus, in the case of the failure of a line of powertaps that is arranged perpendicular to the row of modules, the failurecan be spread across multiple applications. As a result, for serviceprovisioning, it is more likely that service for each application willbe slightly impacted rather than having a catastrophic failure ofservice for one particular application.

Assuming that the rack-mounted computers in a row of modules has similarfunctionality, e.g., the rack-mounted computers function as searchrequest servers, then the data lines can run parallel to the rows 102.However, this is not required, i.e., the data lines could also runperpendicular to the rows 102 of modules.

In general, any portion of the power, cooling or data grids could besupported from either the floor or ceiling. The raised floor 14 (andassociated plenum 22) could be absent, and portions of grids describedas running through the plenum could simply run along floor. Similarly,the suspended ceiling could be absent, and portions of grid described asin the suspended ceiling could be supported by or from the ceiling 16.There can be additional pumps or flow control devices; for example,there can be a pump for each manifold, or a pump at each tap. The powerbusbar 32, manifolds 52/62 and data cabling 72 are shown as linear, butcould include turns; the path of the data cabling 72 and manifolds 52/62can be substantially similar to the path of the busbar 32 (e.g., if thebusbar includes a right-turn, the manifolds 52/62 will also include aright-turn, although the location of the turn relative to the busbarcould be offset to maintain spacing for the module or provide uniformspacing between busbar and manifolds (as in the embodiment of FIG. 4)).A coolant other than water could be used.

There are several potential advantages of the facility 10. Since acomputer module can be moved into place and then connected to nearbytaps with flexible hoses, power cables and data cables, installation ofthe computer module is very simple and can be accomplished by thepersonnel of the operator of the data center without further need forcontractors (specifically plumbing or electrical contractors). Inaddition, once a computer module is connected, it can begin operationessentially immediately. Thus, the data center can begin operating evenif only part of the available space is used. Furthermore, in the eventof maintenance or malfunction, the system of parallel power busbars andcooling manifolds enables the rack-mounted computers attached to onepower and cooling line to be taken off-line while rack-mounted computersin other portions of the data center continue functioning.

A number of embodiments of the invention have been described.Nevertheless, it will be understood that various modifications may bemade without departing from the spirit and scope of the invention.Accordingly, other embodiments are within the scope of the followingclaims.

1. A facility, comprising: one or more enclosures defining an interiorspace; a plurality of horizontally extending power lines, each of thepower lines including a plurality of power taps disposed along a portionof the horizontally extending power line in the interior space; aplurality of horizontally extending coolant supply lines, each of thecoolant supply lines including a plurality of coolant supply tapsdisposed along a portion of the horizontally extending coolant supplyline in the interior space; and a plurality of horizontally extendingcoolant return lines, each of the coolant return lines including aplurality of coolant return taps disposed along a portion of thehorizontally extending coolant return line in the interior space, theflow capacity of the supply taps and the return taps being approximatelyequal; wherein the portions of the plurality of horizontally extendingpower lines having the power taps extend perpendicular to the portionsof the plurality of horizontally extending coolant supply lines havingthe coolant supply taps and the portions of the plurality ofhorizontally extending coolant return lines having the coolant returntaps, and wherein the plurality of coolant supply taps and plurality ofcoolant return taps are spaced at a substantially uniform equal densityacross the interior space.
 2. The facility of claim 1, wherein thesupply taps and return taps are connected to a heat exchanger or acooling plant.
 3. The facility of claim 1, wherein there is no localstorage or buffering of a coolant in the interior space.
 4. The facilityof claim 1, wherein the plurality of power taps are distributed alongeach of the power lines with a substantially regular first spacing in afirst direction and the power lines are spaced with a substantiallyregular second spacing in a second direction perpendicular to the firstdirection, the plurality of supply taps are distributed along each ofthe coolant supply lines with a substantially regular third spacing inthe second direction and the coolant supply lines are spaced with asubstantially regular fourth spacing in the first direction, and theplurality of return taps are distributed along each of the coolantreturn lines with a substantially regular fifth spacing in the seconddirection and the coolant return lines are spaced a substantiallyregular sixth spacing in the first direction.
 5. The facility of claim4, wherein the third spacing is approximately equal to the fifth spacingand the fourth spacing is approximately equal to the sixth spacing. 6.The facility of claim 1, further comprising: a plurality of powerdelivery busbars to provide power to the plurality of power taps, thebusbars defining the power lines, a plurality of coolant supplymanifolds to provide coolant to the plurality of supply taps, thecoolant supply manifolds defining the coolant supply lines, and aplurality of coolant return manifolds to return coolant from the returntaps, the coolant return manifolds defining the coolant return lines. 7.The facility of claim 1, wherein the coolant supply lines and thecoolant return lines are uniformly spaced with a first pitch.
 8. A datacenter, comprising: one or more enclosures defining an interior space; aplurality of horizontally extending power lines, each of the power linesincluding a plurality of power taps in the interior space; a pluralityof horizontally extending coolant supply lines, each of the coolantsupply lines including a plurality of coolant supply taps in theinterior space; a plurality of horizontally extending coolant returnlines, each of the coolant return lines including a plurality of coolantreturn taps in the interior space; and a plurality of clusters ofmodules in the interior space, each cluster located in a spatiallycontiguous area horizontally separated from another cluster by an accessaisle, each module including a plurality of rack-mounted computersconnected to a power tap of the plurality of power taps adjacent themodule and a cooling coil to remove heat from air near the rack-mountedcomputers, the cooling coil fluidly connected between a supply tap ofthe plurality of coolant supply taps and a return tap of the pluralityof coolant return taps adjacent the module; wherein each clusterincludes two or more modules, and wherein each of the two or moremodules is connected to different ones of the plurality of power linesand different ones of the plurality of coolant supply lines and theplurality of coolant return lines.
 9. The data center of claim 8,wherein substantially all of the rack-mounted computers of a particularcluster are dedicated to the same application.
 10. The data center ofclaim 8, wherein the rack-mounted computers of at least two differentclusters of the plurality of clusters are dedicated to differentapplications.
 11. The data center of claim 8, wherein the plurality ofpower taps are distributed with a substantially regular spacing in theinterior space, the plurality of coolant supply taps are distributedwith a substantially regular spacing in the interior space, and theplurality of coolant return taps are distributed with a substantiallyregular spacing in the interior space.
 12. The data center of claim 8,wherein modules are arranged in substantially linear rows.
 13. The datacenter of claim 12, wherein the power lines, coolant supply lines andcoolant return lines are substantially linear.
 14. The data center ofclaim 13, wherein the linear rows of the modules are perpendicular tothe power lines, coolant supply lines and coolant return lines.
 15. Thedata center of claim 14, wherein the power lines are uniformly spaced.16. The data center of claim 15, wherein the power lines are spaced witha pitch approximately equal to a spacing between modules in the rows.17. The data center of claim 8, wherein the power taps are disposed withsubstantially uniform first spacing along each of the power lines, thesupply taps are disposed with substantially uniform second spacing alongeach of the coolant supply lines, and the return taps are disposed withsubstantially uniform third spacing along each of the coolant returnlines.
 18. The data center of claim 17, wherein the second spacing isapproximately equal to the third spacing.
 19. The data center of claim8, wherein the coolant supply lines and the coolant return lines areuniformly spaced with a first pitch.